Chapter 12: Troubleshooting: Production Diagnostics, Common Pitfalls, and Failure Modes

Deploying hardened LangChain systems to enterprise production introduces failure modes spanning authentication boundaries, policy evaluation timeouts, vector retrieval pruning, and agent reasoning loops. When policy enforcement is strictly fail-closed, an upstream misconfiguration or unexpected token format halts execution rather than failing silently.

This chapter provides a concise diagnostic reference for production operations. Using our Enterprise Incident Assistant demonstrator as a concrete case study, we examine real-world failure patterns and demonstrate how to diagnose and resolve them across your own enterprise AI applications.

Exception Taxonomy & Diagnostic Reason Codes

The security extension intercepts operations across multiple Policy Enforcement Points (PEPs) and raises typed exceptions derived from SecurityAuthorizationError. Every denial carries a standardized diagnostic_reason to facilitate immediate troubleshooting:

1. Identity & Context Propagation Pitfalls

Symptom: Requests Fall Back to Anonymous (Clearance 0, Deny-All)

Authorized users suddenly receive AgentAccessDeniedException or queries return zero vector results because the retriever injected the unauthenticated sentinel: {"sec:tenant_id": "__DENY_ALL__"}.

• Root Cause: Invoking downstream chains or secondary agents without passing LangChain's native RunnableConfig drops caller identity, defaulting execution to SecurityIdentity.anonymous().
• Remediation: Always propagate identity explicitly inside the execution configuration:

  You can inspect how identity is bridged into run_config and passed to downstream agents in app/routers/tickets.py of the assistant.

  SECURITY LIBRARY SNIPPET (INFORMATIVE) langchain_security_extension/security_core/identity.py

  PYTHON

  # Correct pattern: forward active identity via RunnableConfig
  run_config = IdentityBridge.create_runnable_config(caller_identity)
  remediation_agent.invoke(inputs, config=run_config)

2. Policy Decision Point (PDP) & Circuit Breaker Failures

Symptom: Intermittent Failures with "PDP circuit breaker is OPEN"

The application abruptly halts all tool and agent executions with fail-closed rejections, even for privileged administrators.

• Root Cause: When security.pdp.mode=OPA is enabled, remote HTTP latency spikes exceeding security.pdp.opa.timeout (default: 50ms) cause repeated evaluation timeouts, tripping the circuit breaker threshold. When tripped to OPEN, the adapter marks sec:circuit_breaker_state = "OPEN" in request configuration and appends [EVENT_CIRCUIT_BREAKER_OPEN | EVENT_CIRCUIT_BREAKER_TRANSITION | LOCAL_PDP_FALLBACK] to diagnostic reasons, enforcing fail-closed rejection.
• Enterprise Best Practice:
  • Structured Error Classification: Catch sec:circuit_breaker_state == "OPEN" or circuit breaker diagnostic markers in the agent telemetry helper to return specialized status BLOCKED_CIRCUIT_BREAKER with HTTP 403 instead of generic 500 server crashes.
  • Enable Local Safety Fallback: Configure the remote adapter with fallback_engine=base_engine. When the remote OPA cluster is partitioned, the adapter safely falls back to local in-process RBAC/CEL rules rather than halting business operations.
  • Tune Thresholds: For distributed clusters, set security.pdp.circuitbreaker.threshold=3 and increase timeout to 0.1 (100ms) to accommodate temporary network jitter.

Symptom: Stale Authorization After Identity or Role Revocation

• Root Cause: The Two-Tier Decision Cache retains static decisions in Tier 2 (L2) for the duration of security.cache.l2.ttl (default: 60s). If an IAM role is revoked, cached decisions allow unauthorized actions until expiry.
• Remediation: Wire your enterprise IAM / SCIM revocation events directly to PDP cache invalidation:

  You can inspect the full implementation in app/security/user_registry.py and app/security/security_integration.py of the assistant.

  DEMO APP (SNIPPET) app/security/user_registry.py

  PYTHON

  # Evict subject immediately upon permission change
  from app.security.security_integration import get_opa_cache_invalidation_handler
  get_opa_cache_invalidation_handler().on_invalidation_event({"subject_id": user_email})

3. Knowledge Ingestion & Vector Retrieval Traps

Symptom: Ingestion Batch Aborts Atomically

• Root Cause: In STRICT_FAIL_CLOSED mode, SecureVectorStoreIngestor rejects the entire batch if a single chunk lacks mandatory metadata (e.g., sec:tenant_id or sec:clearance).
• Remediation: For bulk backfills of legacy documentation, switch temporarily to EVERYTHING_CLASSIFIED (quarantine mode). Unlabelled chunks are automatically stamped with sec:quarantined = True and elevated to RESTRICTED, allowing the batch to complete while keeping unclassified data isolated from regular users.

Symptom: Top-K Starvation in RAG Pipelines

The vector store contains matching runbooks, but the retriever returns only 1 or 2 chunks instead of the requested k=5, causing the LLM to hallucinate or state that it lacks sufficient context.

• Root Cause: Vector similarity search retrieves top-K raw candidates, but post-retrieval in-memory pruning drops chunks whose classification rank exceeds caller clearance or whose department tags mismatch.
• Remediation: Configure candidate oversampling via security.retriever.oversampling (e.g., 3.0 to 5.0). SecureRetriever fetches k * oversampling_factor candidates from the vector database so sufficient valid records survive security pruning.

Symptom: Chunks Rejected with "INVALID_HMAC"

• Root Cause: The HMAC signature (sec:envelope_hash) fails cryptographic verification because security.hmac.secret was rotated without re-indexing, or an external script modified chunk metadata directly in Chroma/Pinecone.
• Remediation: Never update vector metadata out-of-band. Always update documents through SecureVectorStoreIngestor to ensure HMAC signatures remain consistent.

4. Tool Interception, CEL Rules, and Agent Looping

Symptom: CEL Denials with "FAIL_CLOSED_NULL_ENTITY"

• Root Cause: A CEL expression checks an entity attribute (e.g., entities.server[args.server_name].ownerDepartment == subject.dept), but EntityResolver.resolve_entity() returned None because the entity key was invalid or missing. The engine safely fails closed.
• Remediation: Ensure your EntityResolver differentiates between missing entity data (raise an informative application error) and unauthorized access.

Symptom: Runaway Agent Terminated with "TOOL_RETRY_BUDGET_EXCEEDED"

• Root Cause: When a tool access attempt is denied, models configured with weak refusal prompts frequently enter infinite loops, retrying the denied tool with minor parameter mutations until exhausting security.retry.maxViolations (default: 3).
• Remediation:
  • Use on_denial="MODEL_FEEDBACK" for informational tools to provide clear refusal reasoning back to the model context.
  • Reserve on_denial="HARD_ABORT" strictly for state-altering, mutative actions (e.g., server reboots, firewall rule updates).
  • Ensure the agent's system prompt instructs the model to acknowledge policy denials and explain restrictions to the user rather than re-attempting blocked tools.

5. Telemetry Backpressure & Sanitization Over-Redaction

Symptom: LLM Receives Mangled Prompts or Broken Code

• Root Cause: Setting security.pii.redaction.scope=full_in_flight applies destructive one-way redaction across inbound user prompts, retrieved RAG chunks, and tool outputs. Aggressive NER or regex patterns replace technical identifiers, IP addresses, or code snippets with [REDACTED_PII] before the model processes them.
• Remediation: Keep security.pii.redaction.scope=audit_only for internal applications. This ensures prompts and tool outputs reach the LLM uncorrupted, while redacting sensitive tokens strictly in audit logs and SIEM exports.

Symptom: High Volume of "DROP_OLDEST_WITH_METRIC" SIEM Alerts

• Root Cause: The asynchronous audit ring buffer reached capacity because the remote SIEM endpoint (e.g., Splunk HEC) is experiencing network latency or rate-limiting. To preserve inference latency, the daemon worker thread drops oldest non-critical events.
• Remediation: Check network connectivity to your SIEM endpoint, verify HEC authentication tokens, and increase buffer capacity if your application handles bursty traffic.

Production Health Check Checklist

Before promoting any LangChain-based service to production, verify these core security invariants:

For complete property specifications, default thresholds, environment variable overrides, and component mappings, consult Chapter 13: Unified Configuration Reference.